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1.0  INTRODUCTION 


This  plan  documents  the  Joint  Integration  Test  Facility  (JITF)  Program  Manager’s  (PM’s) 
strategy  for  conducting  performance  measurement.  Performance  measures  quantitatively 
provide  important  information  about  products,  services,  and  the  processes  that  produce  them. 
This  information  is  necessary  to  make  intelligent  decisions  regarding  the  JITF  program  and 
assists  the  PM  in  meeting  goals  and  objectives. 

The  JITF  PM  recognizes  that  there  is  increased  attention  on  program  performance  and  results. 
As  the  Department  of  Defense  (DoD)  and  the  Intelligence  Community  shift  from  managing 
acquisition  to  managing  investment,  the  JITF  program  manager  must  demonstrate  accountability 
and  responsibility  for  program  outcomes.  The  Clinger— Cohen  Act  of  1996,  which  was 
previously  known  as  the  Information  Technology  Management  Reform  Act  (ITMRA),  mandates 
this.  The  Clinger-Cohen  Act  states:  • 

“The  DoD  must  improve  day-to-day  mission  processes  and  properly  use 
information  technology  to  support  those  improvements.  Technology  must  be 
fielded  in  an  orderly,  fast  and  efficient' way.  ITMRA  prescribes  an  information 
technology  investment  portfolio  concept,  which  emphasizes  the  need  to  do  a  better 
job  of  prioritizing  information  technology  capital  investments  and  being 
accountable  for  results.  This  applies  from  each  person  individually  up  to  mission 
commanders  and  Congress. " 

In  every  instance,  the  JITF  Performance  Measurement  Plan  attempts  to  align  performance 
metrics  with  Department  of  Defense  Intelligence  Information  Systems  (DoDIIS)  Management 
Board  (DMB)  mission/goal  statements  and  the  Intelligence  Community  Information  Systems 
Strategic  Plan.  The  purpose  of  the  JITF  Performance  Measurement  Plan  is  to  demonstrate 
effectiveness  of  the  process  in  supporting  the  missions  of  both  customer  and  end-user 
organizations.  The  plan  provides  the  basis  for  programmatic  justification  while  at  the  same  time 
developing  the  data  with  which  to  make  both  process  design  and  technology  decisions.  The  plan 
becomes  an  integral  mechanism  to  assist  all  project  stakeholders  in  the  management  of  program 
resources  and  the  prioritization  of  requirements  and  corresponding  system  design. 


1.1  SCOPE 

The  Department  of  Defense  Guide  for  Managing  Information  Technology  (IT)  as  an  Investment 
and  Measuring  Performance  describes  three  distinct  levels  of  management,  each  requiring 
different  information  to  make  an  informed  decision.  These  levels  are: 

□  Enterprise  •  -.Q  Functional'  .  □  Program/Project.  •  • 
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The  units  of  measure  for  these  levels  interconnect  to  form  a  pyramid  (Exhibit  1).  Program  level 
measures  start  at  the  base.  These  are  measures  of  individual  units  of  products  and  individual 
elements  of  service.  These  measures  focus  on  activity  and  task  information  and  workplace 
results.  They  provide  data,  which  is  used  for  tactical  and  execution  management. 

Detailed  information  collected  at  the  program  level  is  used  at  the  functional  level  for  integration 
and  planning,  and  is  used  at  the  enterprise  level  for  policy  and  mission  decisions  and  strategies. 
Information  traveling  up  through  the  levels  of  management  aligns  goals  and  objectives  and 
ensures  that  strategic  initiatives  are  met. 

Compliance  with  Clinger-Cohen  facilitates  the  availability  of  critical  performance  information  at 
all  levels  of  management,  from  the  project/program  to  the  functional  level  through  the  enterprise 
level.  For  JITF,  the  project/program  level  includes  497th  Intelligence  Group  DoDIIS  Executive 
Agent  (DExA)  for  Test  and  Evaluation  (T&E),  Air  Force  Research  Laboratory  (AFRL) 
Information  Handling  Branch  (tFEB)  and  JITF  personnel.  The  functional  level  includes  the 
DMB  and  its  individual  members  (such  as  Defense  Intelligence  Agency  (DIA),  National  Security 
Agency  (NSA),  National  Imagery  and  Mapping  Agency  (NIMA),  Defense  Information  Systems 
Agency  (DISA),  and  the  armed  services).  The  JITF’s  enterprise  level  consists  of  the  Intelligence 
Community  and  DoDIIS. 

Exhibit  1 .  Basic  Feedback  Loop  for  JITF  Levels  of  Performance  Measurement 


The  JITF  Performance  Measurement  Plan  focuses  on  the  project  level.  While  this  information 
will  be  of  interest  to  the  functional  and  enterprise  level  managers,  it  does  not  attempt  to  identify 
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how  this  measurement  information  might  be  applied  at  higher  levels.  The  performance  measures 
proposed  by  this  plan  will  be  essential  to  the  JITF  PM  in  determining: 

□  If  program  goals  are  being  met. 

□  If  JITF  customers  are  satisfied. 

□  If  processes  are  under  control. 

□  Where  improvements  are  necessary. 

□  Input  for  JITF  and  IFEB  Budgetary  decisions. 

□  Requirements  prioritization. 

1.2  PLAN  ORGANIZATION 

Information  in  this  JITF  Performance  Measurement  Plan  is  organized  into  the  following  sections. 
Section  7  identifies  the  scope  of  the  document. 

Section  2  identifies  planning  and  guidance  documents  that  define  how  the  JITF  program 
relates  to  the  Intelligence  Community,  of  which  the  DMB  and  its  participating  members 
are  a  part. 

Section  3  provides  an  overview  of  the  performance  measurement  process.  . 

Section  4  provides  detailed  descriptions  of  the  initial  JITF  performance  measures. 

Section  5  describes  implementation  criteria. 

Section  6  identifies  possible  future  activities  that  can  be  incorporated  into  the  JITF 
Performance  Measurement  Plan. 

Section  7  provides  a  brief  summary. 

Section  8  provides  a  detailed  acronym  listing. 
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2.0  DIRECTIVES  GUIDING  JITF  MISSION 


Enhanced  performance  begins  with  effective  planning  (Exhibit  2).  As  the  Functional  Managers, 
the  DMB,  the  497lh  Intelligence  Group,  and  the  Information  Directorate  of  the  Air  Force 
Research  Laboratory  guide  program  planning  for  JITF.  As  the  executive  agent,  the  497th 
Intelligence  Group  oversees  requirements  for  the  program.  JITF  is  a  critical  component  of 
DoDIIS.  The  DMB,  along  with  representatives  from  the  services,  DISA,  NIMA,  NS  A,  and  DIA, 
provide  system-level  requirements  for  programs  operating  at  DoDIIS  sites.  Managers  at  NIMA 
and  DIA,  with  responsibility  for  DoDIIS,  align  strategic  plans  to  ensure  Intelligence  Community 
requirements  are  met.  This  allows  JITF  to  focus  on  common  goals  and  objectives  in  support  of  a 
diverse  customer  base  comprised  of  intelligence  analysts  from  DIA,  the  Air  Force,  Army,  Navy, 
and  Marines. 


Exhibit  2.  JITF  Hierarchy 
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sp.  PROJECT/PROGRAM 


2.1  DODIIS  MISSIONS 


DoDIIS  mission,  paraphrased,  is  to: 

Provide  the  right  information  at  the  right  time  to  the  right  people. 

To  fulfill  this  mission,  the  JITF  supports  operational  users,  Intelligence  Mission  Application 
(IMA)  program  managers,  and  the  DMB,  through  installation,  integration,  and  infrastructure 
compliance  testing  and  evaluation  as  defined  by  the  IMA  certification  process.  By  performing 
this  function,  the  Intelligence  Community  receives  the  highest  quality  products  to  process 
information. 

The  level  of  testing  performed  by  the  JITF  verifies  installation  procedures,  identifies  resource 
conflicts,  and  assesses  the  operational  impacts  of  applications  functioning  in  a  common  DoDIIS 
environment  defined  (currently)  as  Client  SefveT  Environment  (CSE).  In  the  very  near  future, 
the  common  environment  will  be  Defense  Intelligence  Infrastructure  (DII)  /  Common  Operating 
Environment  (COE).  DII  COE  will  provide  a  common  set  of  infrastructure  services  to 
applications,  a  common  security  environment  for  both  sites  and  applications,  and  a  common  set 
of  management  tools  for  system  administration  and  security  management  fcr  DoDIIS.  DoDIIS 
stands  out  as  a  leader  in  the  community  by  initiating  common  infrastructures  (e.g.,  CSE)  many 
years  ago.  DoDIIS  will  continue  this  evolutionary  path  as  DII  COE  matures  to  meet  DoDIIS 
requirements.  The  JITF  evaluates  the  capability  of  the  IMA  to  operate  in  an  environment  in 
which  computing  resources  including  processors,  configuration 'files,  networking  facilities,  and 
administration  facilities,  are  shared  by  many  systems.. The  JITF  evaluates  the  installation  and 
operation  of  an  IMA  for  its  ability  to  function  in  the  shared  computing  environment  without 
affecting  other  applications  that  are  operating  simultaneously.  In  addition;  JITF  personnel  and 
facilities  support  Joint  Interoperability  Test  Command  (JITC)  interoperability  testing  and 
security  certification  testing  in  conjunction  with  JITF  integration  testing. 

The  JITF  is  located  at  Air  Force  Research  Laboratory/Rome  Research  Site  and  is  a  program 
management  office  staffed  by  a  combination  of  government  employees  and  contractors.  The 
name  JITF  represents  the  actual  facility  used  to  conduct  IMA  integration  testing  in  addition  to 
being  the  designation  for  the  program  office.  JITF  is  also  used  to  globally  describe  the 
integration  testing  process  conducted  in  support  of  the  IMA  certification  process.  JITF  testing  is 
comprised  of  multiple  test  activities  that  reduce  the  risk  of  integrating  new  applications  into 
existing  environments. 


2.2  GUIDANCE  DOCUMENTATION  AND  RESOURCES 

The  following  table  identifies  documents  and  publications  that  are  related  to  JITF  performance 
measurement  activities.  These  guidance  documents  contain  standards  that  are  extracted  by  the 
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DExA  for  Test  and  Evaluation  and  translated  into  requirements  and  processes  that  must  be  met 
(e.g.,  development  and  architectural  requirements,  technology  baselines  including  the  use  of 
Commercial  Off-the-Shelf  [COTS]  products,  IMA  certification  process). 


Table  1.  Guidance  and  Resource  Documents 


Source/Date 

Document  Title  | 

USAF,  4/13/98 

“United  States  Air  Force  Strategic  Plan:  Volume  2  -  Performance  Plan” 

AFRL-RRL, 

3/1/1999 

“Common  User  Baseline  for  the  Intelligence  Community  (CUBIC)  Configuration 
Management  (CM)  Plan,  Version  2.0”  (DRAFT) 

DISA,  7/1/1997 

“Defense  Information  Infrastructure  (Dll)  Common  Operating  Environment  (COE) 
Integration  and  RunTime  Specification  (l&RTS),  Version  3.0” 

DMB,  4/1/1999 

“Department  of  Defense  Intelligence  Information  System  (DoDIIS)  Instructions” 

DISA,  5/28/1 998 

“DoD  Joint  Technical  Architecture  (JTA),  Version  2.0” 

U.S.  Congress, 
1993 

“Government  Performance  and  Results  Act  (GPRA)” 

ISB,  8/1997 

“Intelligence  Community  Information  Systems  Strategic  Plan 
(1999-2003)” 

AFRL-RRS, 

4/4/1996 

“JITF  Concept  of  Operations” 

AFRL-RRS, 

4/21/1999 

“Joint  Integration  Test  Facility  (JITF)  DoDIIS  Integration  Requirements  and  Evaluation 
Procedures,  Version  2.0” 

JCS,  7/1997 

“Joint  Vision  201 0  “ 

DMB, 

11/19/1995 

“Memorandum  of  Agreement  Between  the  Joint  Interoperability  Test  Command 
(JITC),  DoD  Intelligence  Information  Systems  (DoDIIS)  Management  Board  (DMB), 
the  DoDIIS  Executive  Agent  (DExA)  for  Migration  Systems  Test,  and  the  DoDIIS  Joint 
Integration  Test  Facility  (JITF);  SUBJECT:  Interoperability  Test  and  Certification  of 
DoDIIS  Migration  Systems” 

U.S.  Congress, 
1996 

“National  Defense  Authorization  Act  FY  1996,  Division  E:  Information  Technology 
Management  Reform”  (also  known  as  the  “Information  Technology  Management 

Reform  Act  (ITMRA)” 

OUSD-AT, 

JLC-JGSE, 

4/17/1998 

“Practical  Software  Measurement:  A  Foundation  for  Objective  Project  Management 
Version  3.1” 

497IG,  4/1/1999 

“Test  and  Evaluation  Policy  for  Department  of  Defense  Intelligence  Information 

Systems  (DoDIIS)  Intelligence  Mission  Applications  (IMA)” 

DoD,  3/15/1996 

DoD  Directive  5000.1,  "Defense  Acquisition” 

DoD,  3/21/1988 

DoD  Directive  5200.28,  “Security  Requirements  for  Automated  Information  Systems 
(AISS)” 

DoD,  9/26/1991 

DoD  Directive  8320.1,  “DOD  Data  Administration" 

DoD,  1/1973 

DoD  Manual  5200.28-M,  “ADP  Security  Manual” 

DoD,  3/1994 

DoD  Manual  8320.1 -M,  “Data  Administration  Procedures” 
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Source/Date 


DoD,  4/1998 


DoD,  3/1996 


DoD,  1/1997 


DoD,  12/1985 


Grady,  Robert  B. 
Prentice  Hall, 

Inc,  1 992 


IEEE,  EIA 


ISO,  IEC 


JITF,  updated  bi¬ 
monthly 


JITF,  updated  bi¬ 
monthly 


JITF,  updated  bi¬ 
monthly 


DIA,  11/1993 


Document  Title 


DoD  Manual  8320.1  -M-1 ,  “Data  Standardization  Procedures”  _ 


DoD  Regulation  5000.2-R,  "Mandatory  Procedures  for  Major  Defense  Acquisition 
Programs  (MDAPS)  and  Major  Automated  Information  Systems  (MAIS)  Acquisition 
Programs”  _ _ 


DoD  Regulation  5200.1 -R,  “Information  Security  Program” _ 

DoD  Standard  5200.28-STD,  “Department  of  Defense  Trusted  Computer  System 

Evaluation  Criteria” _ _ _ _ _ — 

Practical  Software  Metrics  for  Project  Management  and  Process  Improvement 


IEEE/EIA 12207,  “Industry  Implementation  of  International  Standard  ISO/IEC  12207: 
1995:  Information  technology— software  life  cycle  processes” _ 


ISO/IEC  12207,  “Information  Technology— Software  Life-Cycle  Processes _ 


JITF  Intelink  Virtual  Test  Folder  Web  Site  webl.rome.ic.aov/iitf/vtf/vtf.html 


JITF  Intelink  Web  Site  webl.rome.ic.aov/iitf 


JITF  Internet  Web  Site  httD://wwwjf.afrl.af.mil/proarams/iitf/ 


SC-2610-143-93,  “DoDIIS  Developer's  Guide  for  Automated  Information  Systems 
(AIS)  Security  in  DoD  Intelligence  Information  Systems” 


3.0  PERFORMANCE  MEASUREMENT  PROCESS 


The  performance  measurement  process  is  a  series  of  exact  steps,  which  are  implemented  using  a 
“plan,  perform,  measure,  and  improve”  methodology  (Exhibit  3).  Assessing  effectiveness  and 
the  degree  of  improvement  is  critical  to  the  success  of  this  approach. 


Exhibit  3.  Plan,  Perform,  Measure,  Improve  Method 


1.  Establish  performance  goals/standards.  All  performance  measures  are  tied  to  a  predefined 
goal  or  standard. 

2.  Establish  measures.  Identify  individual  measures. 

3.  Identify  responsible  parties.  A  specific  entity  (group  or  individual)  is  assigned  the 
responsibilities  for  each  of  the  steps  in  the  performance  measurement  process. 

4.  Collect  data.  Tools  to  support  data  collection  and  collection  techniques  need  to  be  identified. 
Timeliness  and  accuracy  need  to  be  ensured. 

5.  Analyze  performance.  .Raw  data  are  formally  converted  into  performance  measures. 
Reporting  requirements  are  determined  to  ensure  that  measures  convey  information  in  an 
understandable  form. . 

6.  Compare  performance  to  target.  Determine  if  any  variation  exists  between  performance  and 
.  targets/standards. 
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7.  Identify  corrective  actions.  Depending  on  the  degree  of  variation  some  corrective  action  may 
be  necessary. 

8.  Institute  changes  to  meet  goals.  If  required,  changes  will  be  implemented  to  reach  goals. 
Planning,  performing,  and  measuring  are  only  of  benefit  when  the  organization’s  focus  is  on 
improvement. 

9.  Identify  new  goals/measures,  if  necessary.  Goals  and  standards  need  periodic  reviews  to 
ensure  they  are  in  step  with  technology  shifts  and  to  focus  on  continued  improvement. 


4.0  JITF  GOALS  AND  MEASUREMENTS 


The  initial  performance  goals  for  the  JITF  program  map  to  the  enterprise  and  functional  level 
goals,  objectives,  and  strategic  actions.  The  enterprise  level  is  described  in  the  Intelligence 
Community  Information  Systems  Strategic  Plan.  The  functional  level  is  described  in  Test  and 
Evaluation  Objectives,  from  the  Test  and  Evaluation  Policy  for  Department  of  Defense 
Intelligence  Information  System  (DoDIIS)  Intelligence  Mission  Application  (IMA). 

Enterprise  level  goals  and  objectives  are  displayed  in  the  first  column  of  Table  2.  Goal  1 
(identify  and  provide  information  systems  services  based  on  customer  needs),  Goal  2  (evolve 
toward  a  fully  integrated,  distributed  information  space),  and  Goal  5  (improve  cost-effectiveness 
of  Intelligence  Community  information  systems)  of  the  Intelligence  Community  Information 
Systems  Strategic  Plan  are  mapped  across  the  row  of  Table  2  to  the  specific  measurements  of 
this'  performance  plan  in  column  4.  '  • 

Functional  level  goals  and  objectives  are  displayed  in  the  second  column  of  Table  2. 
Performance  goals  and  objectives  defined  in  Section  2,  Test  and  Evaluation  Objectives,  from  the 
Test  and  Evaluation  Policy  for  Department  of  Defense  Intelligence  Information  System  ( DoDIIS ) 
Intelligence  Mission  Application  (IMA)  are  mapped  across  the  row  of  Table  2  to  the  specific 
measurements  of  this  performance  plan  in  column  4. 

Project  level  goals  and  objectives  are  defined  in  this  Performance  Measurement  plan,  Section 
4.0.  The  actual  goal  and  objective  is  defined  within  the  measurement  which  are  displayed  in 
Table  2,  column  4  and  in  the  subsections  of  Section  4.0  of  this  Performance  Measurement  plan. 
All  initial  performance  goals  for  the  JITF  program  are  directly  related  to  the  customers  of  the 
JITF.  Table  2  provides  a  cross-reference  of  applicable  enterprise,  functional,  and  project  goals 
and  objectives  to  JITF  performance  measures. 

The  JITF’s  ability  to  perform  testing  and  provide  technical  service  to  members  of  the 
Intelligence  Community  and  customers  of  the  JITF  is  critical  to  the  success  of  the  DoDIIS  and 
Intelligence  Community’s  missions. 


Table  2.  Mapping  of  Enterprise  and  Functional  Goals 
to  JITF  Performance  Measures 


Enterprise  Level 
Goals/Objectives 

Functional  Level 
Goals/Objectives 

Project  Level 
Goals/Objectives 

Project  Level 
Goals/Measures 

Intelligence  Community 
Information  Systems  Strategic 
Plan,  Goal  1  -  Identify  and 
provide  information  systems 
services  based  on  customer 
needs.  Objective  1C  - 

Test  and  Evaluation  Policy  for 
Department  of  Defense 
Intelligence  Information 

Systems  (DODIIS)  Intelligence 
Mission  Applications  (IMA), 
Section  2.  Test  and  Evaluation 

Maximize  Customer 

Satisfaction. 

4.1.1  Customer  Surveys 

4.1.2  Test  Report  Timeliness 

Monitor,  measure,  and 
evaluate  mission  benefits  and 
customer  satisfaction  obtained 
from  Community-level 
investments  in  information 
systems,  services,  and 
technology. 

Objectives:  Ensure  a 

thoroughly  planned, 
understood,  documented, 
comprehensive,  and 
consistent  test  program  to  fully 
test  and  validate  the  DoDIIS 

IMA  in  support  of  DMB 
milestone  decisions  and  user 
needs. 

Intelligence  Community 
Information  Systems  Strategic 
Plan,  Goal  2  -  Evolve  toward 
a  fully  integrated,  distributed 
information  space.  Objective 

2E  -  Migrate  to  a  Community¬ 
wide  information  processing 
environment. 

Test  and  Evaluation  Policy  for 
Department  of  Defense 
Intelligence  Information 

Systems  (DODIIS)  Intelligence 
Mission  Applications  (IMA), 
Section  2.  Test  and  Evaluation 

Increase  IMA  quality. 

4.2.1  Requirements  Met 

4.2.2  Specific  Requirements 

Not  Met 

Objectives:  Determine  and 

document  the  degree  to  which 
IMA  software  conforms  and 
performs  to  established 
standards  and  integration 
requirements,  providing 
sufficient  detail  to  allow 
assessment  of  the  risk  of 
integrating  applications  into 
the  existing  and  planned 
infrastructures  and  platforms. 
Determine  if  the  IMA  software 
meets  requirements  mandated 
in  the  DoDIIS  Instruction. 

Identify  and  document  for 
resolution,  instance  of 
duplicate  functionality  within 
the  IMA  as  mandated  by  the 
DoDIIS  Instructions. 

i 

Intelligence  Community 
Information  Systems  Strategic 
Plan,  Goal  5-  Improve  cost- 
effectiveness  of  Intelligence 
Community  information 
systems.  Objective  5 A  - 
Maximize  the  effectiveness  of 
Community-wide  information 
technology  expenditures. 

No  goals/obiectives  for  cost 
effectiveness  in  DODIIS 
instructions  orT&E  Policvl 

Maximize  efficiency. 

4.3.1  Schedule  Volatility 

4.3.2  Comments  Against  Test 
Reports 

4.1  GOAL  1  -  MAXIMIZE  CUSTOMER  SATISFACTION 


Description  -  The  JITF  exists  to  support  the  user  by  improving  the  effectiveness  of  IMA  use  in 
complex  operating  environments.  Focusing  on  customer  satisfaction  leads  to  an  accepted, 
successful  product  and  service.  Since  traditional  methods  of  assessing  success  are  not  available 
(such  as  market  share  and  growth),  communication  between  the  JITF  and  all  classes  of 
customers  is  critical  to  determining  the  value  of  the  services  and  products  provided  by  the  JITF. 
Customers  of  the  JITF’s  products  and  services  are  varied.  The  following  is  a  partial  list  of  the 
type  of  customers  served  by  the  JITF: 

□  Enterprise  Managers  -  personnel  charged  with  identifying  IMAs  to  be  tested, 
allocating  resources,  and  defining  test  procedures/requirements. 

□  Project  Managers  -  personnel  charged  with  development  and  fielding  of  the  DoDIIS 
IMA. 

□  Engineering  and  development  staff  -  professional  personnel  charged  with  creating 
and  modifying  the  DoDIIS  IMA. 

□  Managers  of  Users  -  personnel  (Site  Managers,  Military  Commanders  at  all  levels, 
and  Civil  Agency  Managers)  who  oversee  the  use  of  DoDIIS  IMAs. 

□  Users  -  personnel  who  use  a  DoDIIS  IMA  on  a  daily  basis. 

Relationship  of  goal  to  DoDHS/DMB  strategic  plans  -  The  JITF’s  purpose  is  to  ensure  that 
software  implemented  in  the  DODIIS  environment  will  function  with  other  DODIIS  products. 
JITF  personnel  test  and  evaluate  software  for  the  DODIIS  Community  in  order  to  meet  the 
Intelligence  Community  Information  Systems  Strategic  Plan's  goal  of  evolving  toward  a  fully 
integrated,  distributed  information  space.  This,  in  turn,  allows  the  customers  to  use  multiple 
DoDIIS  products  simultaneously.  By  polling  the  customers,  the  JITF  can  demonstrate  the 
effectiveness  of  the  testing  process  in  support  of  the  Intelligence  Community  goals.  Providing 
the  customer  with  timely  test  reports  speeds  the  process  of  deploying  quality  IMA  to  end  users; 
thereby,  increasing  customer  satisfaction. 


4.1 .1  Measure  1  .A.  -  Customer  Surveys 


Responsible  individual  -  Measurement  analyst  and  JITF  staff. 

Performance  measure  description  and  objective  —  Customer  surveys  provide  information  on 
users’  level  of  satisfaction  with  all  aspects  of  the  product  or  services  being  delivered.  This 
information  will  be  used  to  identify  quality  attributes  that  are  important  to  customers,  to  improve 


12 


communication  at  all  levels  and  to  identify  issues  for  resolution.  Surveys  will  focus  on  the 
following  areas: 

□  Information  dissemination 

□  Quality  attributes 

□  Subjective  value 

□  Objective  value 

Data  source  —  Contractor  and  JITF  generated  surveys  distributed  at  conferences,  at  Joint  Test 
Planning  Meetings  (JTPM),  to  customers  at  various  stages  in  the  integration  testing  process,  and 
on  Internet  and  Intelink  web  pages. 

Frequency  of  collection  —  Surveys,  will  be  conducted  continuously,  and  results  will  be  analyzed 
quarterly.  Surveys  will  be  available  on  JITF  Internet  and  Intelink  web  pages  and  will  be  focused 
on  specific  issues  to  reduce  completion  and  analysis  time.  Periodic  assessment  on  the  level  of 
participation  by  users  in  the  survey  process  will  determine  if  increased  or  decreased  frequency  or 
other  methods  or  dissemination  are  warranted.-  ’• 

Standards/targets  -  Targets  for  customer  satisfaction  levels  are  determined  for  this  initial 
sampling  by  the  JITF  Project  Manager.  The  target  levels  for  all  customer  issues  are  to  receive 
not  less  than  90%  satisfaction  rating  in  any  area  in  which  30  or  more  surveys  are  returned.  Areas 
include  requirements  understanding,  responsiveness,  information  dissemination;  and  customer 
relations. 

Rationale  for  standard/target  -  On  a  5  point  scale  (i.e.  1-  extremely  satisfied,  2  -  satisfied,  3  - 
neither  satisfied  or  dissatisfied,  4  -  dissatisfied,  5  -  extremely  dissatisfied),  the  goal  should 
always  be  to  get  a  degree  of  satisfaction  as  opposed  to  indifference  or  dissatisfaction.  The 
requirement  of  30  valid  samples  is  required  to  ensure  minimal  statistical  validity. 

Data  requirements  -  Draft  surveys  will  be  reviewed  by  the  JITF  Project  Manager  before 
distribution.  A  majority  of  survey  results  for  any  given  event  must  be  displayed  graphically. 
This  will  be  considered  when  developing  surveys. 

Assumgtions  -  The  sample  number  for  certain  groups  may  not  reach  a  statistically  significant 
number.  Satisfaction  levels  of  statistically  insignificant  numbers  may  be  used  if  the  number  of 
possible  participants  is  limited  by  function.  For  example,  there  may  only  be  10  PMOs  returning 
an  entrance. survey  per  year,  but  there,  may  be  50  returned  surveys  from  a  DoDIIS  -  wide  event. 
In  these  instances,  the  fact. that  the  number  of  responses  does  not  represent  a  statistically 
significant  number  will  be  prominently  displayed.  The  data  may  be  weighted  depending  on  the 
circumstances  or  characteristics  but  all  data  will  be  considered  in  the  evaluation  process.. 
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4.1 .2  Measure  1  .B  -  Timeliness  of  Test  Reporting 

Responsible  individual  -  Measurement  analyst  and  JITF  staff. 

Performance  measure  description  and  objective  -  This  measure  provides  information  on  the 
responsiveness  of  the  JITF  staff.  Customers  of  the  JITF  require  timely  test  report  completion  in 
order  to  meet  predefined  schedules.  Customer  satisfaction  is  greatly  impacted  by  the  speed  and 
quality  of  test  reporting. 

Data  source  -  Configuration  Management  Database  (CMDB)  and  JITF  test  records. 

Frepuencg  of  collection  -  Monthly,  reported  quarterly. 

Standards/targets  -  Targets  for  publication  of  the  JITF  Test  Reports  have  been  set  in  the  DoDIlS 
Joint  Integration  Test  Facility  (JITF)  Concept  of  Operations,  dated  4  April  1999.  The  current 
standard  requires  the  JITF  personnel  to  perform  the  following: 

□  Provide  a  complete  test  report  in  draft  form  to  the  PMO  and  the  DExA  for  T&E  not 

later  than  5  working  days  after  testing  completion.-  '  . 

□  Distribute  the  final  test  report  not  later  than  10  working  days  following  completion  of 
testing. 

□  Post  the  test  results  on  the  Intelink  web  site,  “Virtual  Test  Folder”,  not  later  than  10 
working  days  after  release  of  the  final  report  to  the  PMO  (which  is  the  same  as  20 
working  days  after  completion  of  testing). 

Rationale  for  standard/target  -  The  targets  for  this  measurement  were  established  in  the 
documentation  governing  the  operation  of  the  JITF.  The  time  requirements  were  developed 
through  analysis  of  customer  requirements  and  driven  by  management  directives. 

Data  repuirenients  -  The  data  requirements  are  for  each  product  (software,  hardware,  or 
combination)  tested  by  the  JITF.  The  minimum  data  required  for  each  test  are  the  following: 

□  Date  of  test  completion. 

□  Date  of  draft  test  report  delivery  (electronically,  FedEx,  etc.). 

□'  Date  of  final  test  report  delivery.  . 

□  Date  of  test  results  posting  on  the  “Virtual  Test  Folder”. 

Additional  data  may  be  required  based  on  the  circumstance.  Examples  can  be  a  request  for 
extension  by  the  IMA’s  PMO,  notification  of  delay  based  on  unusual  circumstances  (i.e.,  natural 
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disasters,  terrorist  acts,  declaration  of  post/base/Unified  and  Specified  Command  training 
holiday,  etc.),  and  any  other  written  requests  (email  or  memoranda)  to  extend  the  initial 
established  periods. 

Assumptions  -  All  dates  will  be  considered  firm  dates  in  order  to  perform  calculation  for  this 
measurement.  It  is  assumed  that  reports  were  outside  of  required  time  limits  if  the  JITF  did  not 
receive  a  predated  request  for  extension  of  the  due  date  or  “suspense”  date. 


4  2  GOAL  2  -  INCREASE  QUALITY  OF  INTELLIGENCE  MISSION  APPLICATIONS 

(IMA) 


Description  -  The  JITF’s  function  of  integration  testing  is  a  key  component  in  the  Intelligence 
Community’s  goal  of  providing  quality  intelligence  products.  Testing  in  all  stages  of  software 
development  provides  a  two-fold  benefit:  increased  cost  effectiveness  and  increased  product 
quality.  Emphasis  on  compliance  to  directives  prior  to  fielding  an  IMA  saves  the  DOD 
incalculable  resources.  Compliance  with  the  directives  relating  to  the  JITF  mission  causes  the 
PMOs  to  develop  a  quality  product.  The  two  measurements  in  support  of  this  goal  demonstrate, 
first  the  compliance  to- the  directives;  and  second  the  most  common  problems  identified  by  the 
JITF  during  the  testing  process.  By  measuring  these  two  items,  the  JITF  can  coordinate  with  and 
provide  technical  advice  to  PMOs  in  order  to  avoid  the  most  prominent  problems  in  the  IMA 
development.  Each  defect  in  an  IMA  identified  by  the  JITF  enables  the  program  manager  to 
correct  the  software  at  a  single  location  versus  site  assistance  and  trouble-calls  for  a  worldwide 
system.  This  goal  ultimately  relates  to  increasing  customer  and  end  user  satisfaction  by' 
identified  (prior  to  fielding)  defects,  which  must  be  fixed.  In  these  cases,  either  the  defect  is 
repaired  prior  to  release  or  workarounds  are  issued  with  the  product. 

Relationship  o£goal  to  DoDIIS/DMB  strategic  Qians  -  The  JITF  mission  entirely  relates  to  the 
Intelligence  Community’s  Goal  2  -  Evolve  toward  a  fully  integrated,  distributed  information 
space.  Objective  2E  -  Migrate  to  a  Community-wide  information-processing  environment.  The 
DMB  and  the  T&E  DExA  developed  the  following  tasks  for  the  JITF: 

□  to  determine  and  document  the  degree  to  which  IMA  software  conforms  and  performs 
to  established  standards  and  integration  requirements,  providing  sufficient  detail  to 
allow  assessment  of  the  risk  of  integrating  applications  into  the  existing  and  planned 
infrastructures  and  platforms; 

□  to  .determine  if  the  IMA  software  meets  requirements  mandated  in  the  DoDIlS 
Instructions', 

□  to  identify  and  document  for  resolution,  instances  of  duplicate  functionality  within  the 
IMA  as  mandated  by  the  DoDIIS  Instructions.  ■ 
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By  identifying  the  requirements  met  by  the  product  tested,  the  entire  Intelligence  Community  can 
determine  the  quality  of  the  product  for  use.  By  identifying  the  most  frequent  and  prominent 
findings,  the  JITF  can  educate  the  PMOs  and  developers  in  problems  to  watch  for,  to  avoid,  and 
to  correct.  These  steps  increase  the  quality  of  each  product  tested  in  the  JITF. 


4.2.1  Measure  2.  A  -  Requirements  Met 


Responsible  individual  -  Measurement  analyst  and  JITF  personnel. 

Performance  measure  description  and  objective  -  The  Requirements  Met  measure  counts  the 
number  of  defined  requirements  as  defined  in  Entrance  and  Exit  Criteria  for  JITF  Integration 
Testing,  Documentation,  Installation  and  Configuration,  Environment,  Operation,  User  Interface, 
and  Security  requirements  tables.  This  measure  is  an  indication  of  the  software  design  progress. 
It  provides  information  on  the  number  of  requirements  met  and  the  percentage  of  requirements 
met  per  software  application  tested.  Furthermore,  it  demonstrates  the  value  of  independent 
integration,  installation,  and  configuration  testing  of  DODIIS  IMAs. 

Data  source  -CMDB  and- JITF  Testing  Reports. 

Frequency  of  collection  -  Monthly,  reported  quarterly. 

Standards/targets  -  Increase  the  average  percentage  of  requirements  met  by  5%  within  one 
calendar  year  after  implementation  of  this  performance  measurement  plan.  Initial  research  needs 
to  be  conducted  on  the  current  year’s  (FY  1999)  average  percentage  of  requirements  met  in  order 
to  determine  the  baseline  for  the  performance  goal. 

Rationale  for  standard/target  -  This  is  an  initial  target.  The  target  will  be  refined  as  significant 
quality  increases  are  realized.  Once  PMO  incorporate  the  new  processes  into  the  development, 
the  window  for  increased  quality  will  be  limited. 

Data  requirements  —  The  number  met  requirements,  as  defined  in  Entrance  and  Exit  Criteria  for 
JITF  Integration  Testing,  will  be  determined  by  the  test  reports  provided  at  the  end  of  a  test.  A 
percentage  value  will  be  assigned  per  IMA  tested.  Format  of  the  data  may  be  changed  by  the 
JITF  as  required. 

Assuniptions  -  As  the  objectives  of  the  performance  plan  are  realized,  the  target  of  increasing 
the  percentage  of  requirements  met  will  be  adjusted  to  progressively  smaller  percentage 
increases  due  to  the  fact  that  a  target  of  100%  compliance  is  not  realistic. 
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4.2.2  Measure  2.B  -  Requirements  Not  Met 


Responsible  individual  -  Measurement  analyst  and  JITF  personnel. 

Performance  measure  description  and  objective  -  This  measurement  provides  information  on 
the  impact  of  test  findings  and  the  most  frequently  found  test  findings. 

Data  source  -CMDB. 

Frequency  of collection  -  Monthly. 

Standards/targets  -  Initial  research  needs  to  be  conducted  on  the  impact  of  defects  on  the  end- 
users.  For  the  initial  target,  all  defects  identified  in  the  JITF  test  findings  will  be  entered  into  the 
CMDB.  The  target  is  to  present  the  five  most  frequent  test  findings  during  the  JTRR  in  order  to 
reduce  the  frequency  of  the  most  common  defects  by  5%  within  a  twelve-month  period 
following  the  findings  initial  identification  and  presentation  in  JTRR  format. 

Rationale  for  standard/target  -  The  JITF’s  primary  goal  is  ensure  the  greatest  customer 
satisfaction  by  providing  the  highest  quality  products  to  end-users.  Reduction  of  the  occurrence 
of  the  most  common  problems  will  reduce  end  user  dissatisfaction. 

Data  requirements  -  Test  findings  are  identified  in  Entrance  and  Exit  Criteria  for  JITF 
Integration  Testing,  Documentation  (DOC-1  through  DOC-29),  Installation  and  Configuration 
(INST-1  through' INST-39),  Environment  (ENV-1  through  ENV-7),  Operation  (OPS-1  through 
OPS-28),  User  Interface  (GUI-1  through  GUI-13),  and  Security  (SEC-1  through  SEC-14) 
requirements  tables:  For  each  IMA  tested,  each  test  finding  will  be  identified  with  its 
requirement  number  (e.g.  DOC-27)  and  its  associated  impact  code.  Test  findings  are  assigned 
impact  codes  to  document  the  severity  of  each  finding.  Impact  codes  are  defined  in  the  CUBIC 
CM  Plan.  The  following  defines  the  four  (4)  types  of  impact  codes  used  by  the  JITF: 

□  Impact  Code  i  -  A  finding  that,  without  resolution,  either 

♦  prevents  the  application  from  proceeding  further  in  testing  or  operation; 

♦  prevents  either  the  application  or  another  application  or  component  of  the 
infrastructure  from  operating  properly; 

♦  creates  a  security  vulnerability  in  the  mission  application  or  site  architecture  that 
can  be  exploited  by  a  general  user  without  taking  advantage  of  other 
vulnerabilities  or  capabilities;  or 

♦  excessively  increases  the  level  of  effort  of  site  personnel  to  install,  manage,  or  use 
the  mission  application  or  other  applications. 

.  □  Impact  Code  2  —  An  urgent  impact  code  that  applies  whenever  significant  mission 
support  degradation  occurs,  or,  if  necessary  to: 
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♦  prevents  the  application  from  proceeding  further  in  its  testing  or  operation; 

♦  has  a  significant  effect  on  the  operation  of  either  the  application  or  on  another 
application  or  component  of  the  infrastructure;  or 

♦  creates  security  vulnerability  in  the  application  or  site  architecture  that  could  be 
exploited  by  a  general  user  only  if  the  user  is  able  to  take  advantage  of  other 
vulnerabilities  or  capabilities  not  typically  available  to  him  or  her. 

□  Impact  Code  3  -  A  finding  that,  without  resolution,  has  a  significant  effect  on  the 
operation  of  either  the  application  or  on  another  application  or  component  of  the 
infrastructure,  the  finding  can  be  temporarily  resolved  by  a  workaround  that  is 
implemented  as  a  change  in  procedure  or  configuration.  The  successful 
implementation  of  the  workaround  does  not  require  technical  expertise  that  is  not 
expected  of  general  users,  or  the  workaround  does  not  cause  significant  level  of  effort 
by  site  administrators.  The  workaround  does  not  cause  significant  delay  in 
integration  testing;  instead,  it  can  be  proposed  and  evaluated’  during  integration 
testing  at  the  JITF. 

□  Impact  Code  4  -  A  finding  that  does  not  prevent  the  application  from  proceeding 
further  in  its  testing  or  does  not  significantly  affect  the  operation  of  the  application  or 
another  application  or  component  of  the  infrastructure.  The  finding  can  be  resolved 
by  a  workaround  that  can  be  implemented  as  a  change  in  procedure  or  configuration 
during  integration  testing  without  a  significant  level  of  effort,  or  the  finding  can  be 
left  as  is.  Even  though  the  finding  has  some  affect  on  the  configuration  or  operation 
of  the  application  or  of  other  components  of  the  site  architecture,  the  general  user  will 
be  able  to  perform  mission  functions,  and  the  administrator' will  be  able  to  manage  the 
application.  Findings,  in  this  category  are  of  lesser  importance,  but  the. accumulation 
of  Impact  4  findings  may  result  in  an  overall  finding  at  a  higher  Impact  level. 

Assumptions  -  The  greatest  assumption  of  this  measurement’s  target  is  that  PMOs  will  heed  the 
advice  of  the  JITF  during  JTRR  and  attempt  to  eliminate  the  most  frequent  findings  from  their 
product  prior  to  testing. 


4.3  GOAL  3  -  MAXIMIZE  EFFICIENCY 


Description  -  As  with  any  organization,  the  JITF  management  tries  to  use  the  funds  allocated 
with  the  most  efficiency.  The  measurements  under  this  section  relate  to  using  people  and  other 
resources  with  the  highest  effectiveness.  The  measurement  concerning  the  schedule  volatility 
relates  to  the  most  efficient  scheduling  of  personnel,  equipment,  and  facilities  in  order  to  avoid 
lag  time  and  overtime.  The  measurement  concerning  the  number  and  type  of  comments  against 
test  reports  relates  to  the  quality  of  the  primary  output  of  the  JITF. 
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Relationship  of  goal  to  DoDHS/DMB  strategic  glans  -  Accurate  scheduling  and  optimizing 
quality  of  test  reporting  all  directly  relate  to  the  Intelligence  Community  Information  Systems 
Strategic  Plan,  Goal  5  -  Improve  cost  effectiveness  of  Intelligence  Community  information 
systems.  Objective  5A  is  to  maximize  the  effectiveness  of  Community-  wide  information 
technology  expenditures.  Funding  of  the  JITF  is  a  community-wide  information  technology 
expenditure. 


4.3.1  Measure  3.A  -  Schedule  Volatility 

Responsible  individual  -  Measurement  analyst  and  JITF  staff. 

Perfonnance  measure  description  and  objective  -  Schedule  variations  are  a  primary  cause  for 
inefficiency.  This  measurement  will  determine  the  average  length  of  time  of  schedule  changes, 
whether  the  dates  were  moved  up  or  back,  and  the  primary  causes  for  volatility. 

Data  source  -  The  schedule  maintained  by  the  JITF  and  CM  and  the  various  documents  (emails, 
memoranda,  and  records  of  phone  conversation)  justifying  schedule  changes. 

Frequency  of  collection  -  Monthly,  reported  quarterly. 

Standards/targets  -  The  initial  data  will  be  collected  during  the  first  performance  plan 
implementation  period.  Initial  research  can  be  done  to  determine  the  average  length  of  time  for 
schedule  changes.  Data  collection  for  the  primary  causes  for  schedule  changes  is  being 
established  during  this  period.  After  this  period  of  baseline  establishment,  the  target  level  will  be 
to  reduce  the  schedule  volatility  by  5%  per  quarter.  Volatility  is  defined  as  the  number  of  days 
the  test  is  moved  (either  further  into  the  future  or  closer  to  the  current  date).  For  example,  if  20 
days  of  change  occurred  in  October  1998,  the  5%  reduction  would  be  realized  if  19  (or  less)  days 
of  change  occurred  in  October  1999. 

Rationale  for  standard/target  -  This  is  an  initial  target.  The  target  will  be  refined  as  significant 
quality  increases  are  realized.  Once  PMOs  and  the  JITF  staff  incorporate  the  new  management 
and  schedule  estimate  processes,  the  window  for  increased  quality  will  be  limited. 

Data  requirements  -  The  following  items  must  be  provided  at  the  beginning  of  the  measurement 
period: 

•  □  The  test  schedule  calendar  for  the  coming  calendar  quarter  will  be  provided  not  later 
than  the  first  day  of  the  quarter.  The  calendars  will  be  provided  for  the  first  quarter  of 
the  fiscal  year  NLT  October  1st,  for  the  second  quarter  NLT  January  1st,  for  the  third 
quarter  NLT  April  1st,  and  for  the  fourth  quarter  NLT  June  1st. 
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□  The  schedule  for  the  coming  month  will  be  provided  not  later  than  the  1st  day  of  the 
month.  For  example,  the  projected  schedule  for  October  1st  through  October  31s' 
must  be  provided  no  later  than  October  1st. 

□  The  actual  occurrences  for  the  previous  month  will  be  provided  no  later  than  the  10lh 
of  the  succeeding  month.  For  example,  all  testing/actions  having  occurred  in  the 
month  of  October  must  be  provided  no  later  than  November  10,h. 

□  Either  copies  of  documentation  of  the  request  for  change  or  a  synopsis  of  each  reason 
for  schedule  change  for  the  previous  month  will  be  provided  with  the  previous 
months  actual  occurrences  no  later  than  the  10,h  of  the  month. 

Assumptions  -  As  the  results  of  the  performance  plan  are  realized,  the  target  of  increasing  the 
percentage  of  requirements  met  will  be  adjusted  to  a  maintenance  level  or  smaller  percentage 
increases  due  to  the  fact  that  a  target  of  100%  or  more  compliance  is  not  realistic. 


4.3.2  Measure  3.B  -  Comments  Against  Test  Report 

Responsible  individual  -  Measurement  analyst  and  JITF  staff. 

Performance  measure  description  and  objective  -  This  measurement  is  to  determine  the  quality 
of  the  primary  output  of  the  JITF.  •  The  number  of  comments  against  the  draft  test  report  will 
show  a  combination  of  typographical  errors  (or  general  word-smithing)  and  technical 
inaccuracies  in  the  report. 

Data  source  -  Document  Review  Reports  against  the  draft  test  reports. 

Frequency  of  collection  -  Monthly,  reported  quarterly. 

Standards/targets  -  No  more  than  3  typographical  (editorial)  comments  and  5  technical 
comments  against  the  draft  test  report.  No  typographical  errors  in  the  final  test  report. 

Rationale  for  standard/target  -  Reduction  of  the  number  of  comments  against  test  reports  will 
reduce  the  amount  of  time  to  produce  the  final  reports;  thereby,  increasing  the  overall 
effectiveness  of  the  test  reporting  process. 

Data  regmreinents  -  The  number  of  comments  against  each  report  documented  within  the 
month  of  reporting. 

Assumptions  -  The  assumption  that  a  technically  and  typographically  correct  test  report  will 
receive  no  comments  against  it  is  unrealistic.  Individuals  reviewing  the  draft  will  continue  to 
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comment  in  order  to  meet  internal  wording  practices;  therefore,  a  100%  reduction  of  all 
comments  is  not  possible. 
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5.0  PLAN  IMPLEMENTATION 


This  plan  implements  Steps  1,  2,  and  3  of  the  performance  measurement  process  detailed  in 
Section  3.  Collection  of  data  (Step  4)  has  begun  for  some  of  the  identified  measures. 


5.1  MEASUREMENT  VALIDATION 


Ensuring  that  the  measures  identified  in  this  plan  are  valid  and  will  provide  needed  information 
for  the  JITF  PM  is  critical  to  the  success  of  the  performance  measurement  process.  Acceptance 
of  this  plan  by  the  JITF  PM  is  the  initial  validation  of  the  selected  measures. 

All  selected  measures  have  been  tied  to  strategic  goals  for  the  organizations  responsible  for  the 
JITF  program.  As  performance  measurement  expands  beyond  the  project  level,  or  as  enterprise 
goals  and  objectives  change,  recurring  validation  of  the  selected  measures  will  also  be  required. 
Fundamentally,  the  six  measures  selected  for  JITF  performance  measurement  are  basic  in  their 
orientation  and  a  majority  should  apply  for  the  duration  of  the  JITF  program. 


5.1 .1  Feedback  -  Check  and  Act 


Performance  measurement  is  designed  to  collect  critical  information  to  allow  management  to 
make  informed  decisions.  This  decision  making  process  creates  a  cultural  focus  on  continual 
improvement  and  also  provides  the  manager  with  justification  for  resource  allocation,  and  go/no- 
go  actions.  Exhibit  4  outlines  the  feedback  loop  designed  to  ensure  the  JITF  PM  takes 
appropriate  corrective  actions  when  necessary,  including  response  to  direction  from 
organizations  outside  the  immediate  JITF  enterprise. 
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Exhibit  4.  Basic  Feed  Back  Loop  for  JITF  Performance  Management 


5.2  DATA  COLLECTION 


All  data  sources  need  to  be  educated  on  the  performance  measurement  process.  This  includes 
members  of  the  JITF,  the  JITF  PM  at  Air  Force  Research  Laboratory,  Rome  Research  Site,  and 
CM  (i.e.,  CUBIC).  A  high  level  of  cooperation  is  required  among  these  groups  and  the 
measurement  analyst  to  ensure  the  receipt  of  accurate  and  timely  data  from  the  source.  It  is 
critical  to  the  success  of  the  JITF  performance  measurement  process  that  all  sources  understand 
that  the  data  collected  will  be  used  to  improve  the  decision  making  process  and  not  to  assign 
blame. 

Before  data  collection  is  begun,  a  clear  understanding  of  the  definitions  associated  for  each 
measure  needs  to  be  reached.  Analysis  and  recommendations  lose  value  when  the  definition  of  a 
measured  item  is  not  clearly  presented  at  the  outset.  An  example  of  this  is  lines  of  code  count  to 
determine  defect  density. 

The  measurement  analyst  will  be  responsible  for  collecting  all  data  from  the  source. 


5.3  DATA  ANALYSIS  TOOLS 


Initial  tools  will  be  simple  to  use,  with  low  learning  curves.  A  database  has-been  developed  to 
track  survey  results.  This  database  will  be  modified  to  accommodate  future  surveys. 
Spreadsheets,  scheduling,  and  word  processing  packages  to  report  results  will  also  be  used.  The 
CMDB  is  also  gonsidered  a  tool  for  use  in  the  performance  measurement  process. 
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Initial  reporting  will  use  Microsoft  Office  products  and  Project  Control  Panel.  Additional  tools 
may  be  added  as  the  amount  of  data  and  the  type  of  metrics  reported  changes. 


5.4  SCHEDULE 


Several  of  the  measurements  included  in  the  plan  are  to  be  collected  over  multiple  test  events.  It 
is  recommended  that  these  measures  be  collected  for  12  months  to  provide  adequate  time  for 
educating  the  JITF  PM  and  related  organizations.  This  will  also  allow  for  the  development  of 
trend  data.  During  that  time,  feedback  and  careful  analysis  of  how  the  performance 
measurement  process  is  working  is  required.  Additional  measures  can  and  should  be  added 
during  this  initial  performance  period.  A  formal  review  of  the  costs  and  benefits  of  the 
performance  measurement  activity  should  occur  at  the  end  of  the  12-month  period. 


5.5  COMMUNICATING  RESULTS 


Whenever  possible,  results  will  be  displayed  in  appropriate  graphical  format,  although  textual 
reports  will  be  required  to  complete  the  analysis  for  various  items.  All  reports  will  be  produced 
quarterly  at  a  minimum,  monthly  if  otherwise  required.  Recommendations  will  be  included  in 
result  presentations.  This  is  also  part  of  the  feedback  and  improvement  process.  Table  3 
identifies  the  anticipated  formats  for  reporting  measurement  results. 


Table  3.  Reporting  Formats 


Measurement  Number  And 
Description 

Anticipated  Format 

1  .A.  Customer  Surveys 

Bar  graph  formats  where  possible. 

Textual  report  for  analysis  of  results  and  recommendations. 

1  .B  Timeliness  of  Test 

Reporting 

Bar  Graph  formats  displaying  the  report  timing. 

Textual  report  for  exceptions  and  waivers  to  the  standards. 

2.A  Requirements  Met 

Bar  Graph  formats  showing  percentages  met  per  product  tested. 

2.B  Requirements  Not  Met 

Bar  Graph  formats  showing  number  of  each  type  of  finding. 

Textual  report  for  descriptions  of  most  common  findings. 

3.A  Schedule  Volatility 

Gantt  chart  for  both  projected  schedules  and  actual  schedules. 

Line  graph  for  showing  variation. 

3.B  Comments  Against  T est 
Report 

Bar  Graph  formats  showing  number  of  comments  per  test  report 
produced. 

Textual  report  for  descriptions  of  most  common  types  of  comments. 
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5.5.1  Measure  1  .A  -  Customer  Surveys 


Customer  Satisfaction  Survey: 
Technical  Assistance 

■  Very 


Satisfaction  Levels: 
Technical  Assistance 


1  2  3 


Question 
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5.5.2  Measure  1  .B  -  Timeliness  of  Test  Reporting 


5.5.3  Measure  2.A  -  Requirements  Met 
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5.5.4  Measure  2.B  -  Requirements  Not  Met 
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GUI  Findings 
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5.5.5  Measure  3.  A  -  Schedule  Volatility 


February  2000  Test  Days 


Testl 


Test2 


Test3 


Diagonal: 

2nd  Date  ? 


Available  Days 
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Days  Changed 


5.5.6  Measure  3.B  -  Comments  Against  Test  Report 


Comments  Against  Test  Reports 


6.0  FUTURE  CONSIDERATIONS 


Most  project-specific  issues  fall  into  one  of  six  common  areas. 

O  Schedule  and  Progress 
O  Resources  and  Cost 
O  Growth  and  Stability 
O  Product  Quality 
O  Development  Performance 
O  Technical  Adequacy 

The  initial  performance  measures  outlined  in  Section  4  touch  on  three  of  these  six  areas: 
Schedule  and  Progress,  Resources  and  Cost,  and  Product  Quality.  Additional  measures  relating 
to  these  three  areas  may  be  taken  as  the  performance  measurement  activity  matures  for  the  JITF 
program.  This  analysis  will  identify  factors  that  contribute  to  the  decision  making  process, 
which  need  to  be  institutionalized  across  the  enterprise.  Identifying  Intelligence  Community 
requirements  will  be  critical  to  the  success  of  the  JITF  performance  measurement  activity  and 
will  ensure  the  Intelligence  Community’s  ability  to  leverage  JITF’s  expertise  in  performance 
measurement  for  future  integration  testing,  and  the  development  of  independent  verification  and 
validation  testing  capability. 

Cost  measurements  that  provide  Return  on  Investment  (ROI)  and  demonstrate  effectiveness  and 
efficiency  within  the  project  will  be  of  interest  to  all  levels  of  the  enterprise.  These  types  of 
measures  will  be  challenging  to  collect,  but  are  necessary  to  justify  budgets  and  demonstrate  the 
value  of  JITF.  Clinger-Cohen  focuses  on  effectiveness  and  efficiency.  Intelligence  Community 
strategic  plan  emphasizes  these  factors  as  well.  It  is  not  possible  to  demonstrate  efficiency  and 
effectiveness  without  calculating  the  costs  and  benefits  of  the  JITF  program.  Future  versions  of 
the  JITF  Performance  Measurement  Plan  will  need  to  include  these  types  of  measures. 

Included  in  the  performance  measurement  activity  is  the  acknowledgement  that  performance 
measurement  itself,  is  a  measurable  process.  Analysis  also  needs  to  be  conducted  on  the  cost  and 
benefits  of  collecting  measures.  This  activity  is  embedded  as  part  of  the  feedback  loop  described 
in  Section  3.1,  but  also  needs  to  be  formalized  and  reviewed  periodically.  Conducting  this 
analysis  will  ensure  that  resources  are  not  expended  on  collecting  data  that  are  not  used  or  add  no 
value  to  the  enterprise. 
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6.1  SUGGESTED  ACTIONS 


The  following  is  a  suggested  list  of  actions  that  might  be  taken  by  the  JITF  PM  to  further 
understanding  of  the  performance  measurement  process  and  increase  the  level  of  sophistication 
and  maturity  of  the  JITF  enterprise. 

□  Attend  training  on  Practical  Software  Measurement  (PSM)  methods.  The  Joint 
Logistics  Commander  (JLC)  Joint  Group  on  systems  engineering  sponsors  PSM.  The 
JLC  is  comprised  of  members  from  each  of  the  services  who  work  on  issues 
applicable  to  all  parts  of  the  DoD. 

□  Coordinate  with  a  software  developer  in  industry  (e.g.  Microsoft)  to  review  their 
software  testing  facilities  for  process  improvement. 

□  To  determine  a  snapshot  of  the  testing  process,  contract  with  a  measurement  company 
(e.g.  GartnerMeasurement)  to  perform  benchmarking  for  comparison  of  the  JITF  to 
similar  corporations  conducting  software  testing. 

□  Coordinate  with  other  military  or  government  organizations  (e.g.  Army  Research 
Laboratory),  which  have  successful  performance  measurement  programs,  to  review 
lessons  learned  and  to  acquire  performance  measurement  process  improvements. 
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7.0  SUMMARY 


The  JITF  PM  is  unique  in  the  intelligence  arena  for  exploring  and  implementing  performance 
measurement.  By  law,  government  organizations  are  required  to  collect  and  act  on 
measurements  designed  to  improve  and  document  performance.  Initial  measurements  are  for 
primary  use  by  the  JITF  PM.  As  further  understanding  of  the  benefits  of  the  measurement 
process  is  gained,  expansions  in  measurement  techniques  and  reporting  will  be  implemented. 
This  will  benefit  all  levels  of  the  JITF  enterprise. 
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8.0  ACRONYM  LIST 


Acronym  Definition 

497IG 

497th  Intelligence  Group 

AFRL 

Air  Force  Research  Laboratory 

CM 

Configuration  Management 

CUBIC 

Common  User  Baseline  for  the  Intelligence  Community 

DCI 

Director  of  Central  Intelligence 

DIA 

Defense  Intelligence  Agency 

DISA 

Defense  Information  Systems  Agency 

DMB 

DoDIIS  Management  Board 

DoD 

Department  of  Defense 

DoDDS 

Department  of  Defense  Intelligence  Information  System 

EIA 

Electronics  Industries  Alliance 

EC 

International  Electrotechnical  Commission 

EEE 

Institute  of  Electrical  and  Electronic  Engineers 

E 

Information  Directorate  of  AFRL 

EE 

Information  and  Intelligence  Exploitation  Division  of  E 

EEB 

Information  Handling  Branch  of  EE 

ISB 

Intelligence  Systems  Board 

ISO 

International  Organization  of  Standardization 

IT 

Information  Technology 

ITMRA 

Information  Technology  Management  Reform  Act 

JCS 

Joint  Chiefs  of  Staff 

JGSE 

Joint  Group  on  Systems  Engineering 

JLC 

Joint  Logistics  Commanders 
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Acronym 


Definition 


NIMA 

National  Imagery  and  Mapping  Agency 

NSA 

National  Security  Agency 

OUSD-AT 

Office  of  the  Under  Secretary  of  Defense  for  Acquisition  and  Technology 

PMO 

Program  Management  Office 

ROI 

Return  on  Investment 

T&E 

Test  and  Evaluation 
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